Relationship between Identification Metrics: Expected Confusion and Area Under a ROC Curve
نویسندگان
چکیده
The mathematical relationship between the expectedconfusion metric and the area under a receiver operating characteristic (ROC) curve is derived. Given a limited database of subjects and an identification technique that generates a feature vector per subject, expected confusion is used to predict how well the feature vector will filter identity in a larger population. Related is the area under a ROC curve that can be used to determine the probability of correctly discriminating between subjects given the feature vector. These two measures have different connotations, but we show mathematically and verify experimentally that a simple transformation can be applied to the expected confusion to find the probability of incorrectly discriminating between subjects, which is the complement of the area under a ROC curve. Furthermore, we show that as a function of the number of subjects, this transformed expected-confusion measure converges more quickly than direct calculation of the area under a ROC curve.
منابع مشابه
Theoretical vs. empirical discriminability: the application of ROC methods to eyewitness identification
Receiver operating characteristic (ROC) analysis was introduced to the field of eyewitness identification 5 years ago. Since that time, it has been both influential and controversial, and the debate has raised an issue about measuring discriminability that is rarely considered. The issue concerns the distinction between empirical discriminability (measured by area under the ROC curve) vs. under...
متن کاملSupport Vector Machines and Area Under ROC curve
For many years now, there is a growing interest around ROC curve for characterizing machine learning performances. This is particularly due to the fact that in real-world problems misclassification costs are not known and thus, ROC curve and related metrics such as the Area Under ROC curve (AUC) can be a more meaningful performance measures. In this paper, we propose a SVMs based algorithm for ...
متن کاملInitial clinical validation of an embedded performance validity measure within the automated neuropsychological metrics (ANAM).
The measurement of effort and performance validity is essential for computerized testing where less direct supervision is needed. The clinical validation of an Automated Neuropsychological Metrics-Performance Validity Index (ANAM-PVI) was examined by converting ANAM test scores into a common metric based on their relative infrequency in an outpatient clinic sample with presumed good effort. Opt...
متن کاملTechnical Note: Towards ROC Curves in Cost Space
ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area under the optimal cost curve. In this note we present some new findings and connections between ROC space and cost space, by using the expected lo...
متن کاملOptimizing Area Under Roc Curve with SVMs
For many years now, there is a growing interest around ROC curve for characterizing machine learning performances. This is particularly due to the fact that in real-world problems misclassification costs are not known and thus, ROC curve and related metrics such as the Area Under ROC curve (AUC) can be a more meaningful performance measures. In this paper, we propose a quadratic programming bas...
متن کامل